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To all whom it may concern: 



Be it known that We, 

Markus ANDREASSON and Per ASTRAND 
have invented certain new and useful improvements in 

METHOD AND DEVICE FOR RECORDING OF INFORMATION 



of which the following is a full, clear and exact description: 
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METHOD DEVICE FOR RECORDING OF INFORMATION 



TECHNICAL FIELD OF THE INVENTION 

The present invention relates to a method and a 
device for recording of LeitL by ituayiny Lhe Le-*.L un an 
optical sensor in a handheld device, said sensor being 
10 intended for* digital recording of images, 

BACKGROUND OF THE INVENTION 

It is sometimes desired to abstract parts of text or 
imaye inluruiaLiun in a document whiuh may latex' be edited 
15 using appropriate software in a computer. A known method 
ul iupuLtiny te.*L and imaye informal iuxi into a computer 
is to use a stationary or portable scanner. 
H A stationary scanner is suitable for entering entire 

f r5i pages with text and image information, the scanner being 

j' s .' 20 automatically passed dCiOaa the paye dL a uuxxstaxxL upeed. 

fj This type of scanner is not suited for inputting selected 

party ul inlormatiou on a paye. 

A portable, handheld scanner may be used any time 
interesting information is to be scanned, but normally 
25 has a limited field of view. 

US-5 , 301, 243 disclose* a hand- held ycanner lux redid- 
v ing characters from a string of characters on a sub- 

strate. The scannex- is moved in contact with the 
substrate along the character line and has an optical 
30 system which images a small part of the substrate- The 
optical system comprises a CCD type line sensor provided 
with a plurality of liqht- sensitive elements arranged in 
a line- When the scanner is passed across the characters 
on the substrate, a succession of vertical slices of the 
35 characters and of the spaces between them is recorded. 

The slices are stored in the scanner as a digital bit -map 
image. OCR software (OCR Optical Character Recognition) 
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is than uR^d to identify fchfi nhararth^-rff and P5hore thpm in 
character- coded form, such as ASCII code. The character 
-re»nngni f.i on nan by msdfi pithpr in thp. flnannfir n*r in an. 
external computer to which the bit-map image is sent. 
5 Another type of hand-held scanner tor inputting text 

is disclosed in 113-4,949,331. This scanner has a two- 
dimensional sensor which records images of the underlying 
surface as the seamier is being moved across the same. 
The scanner is restricted to movements in a direction 
10 which is determined by a wheel in contact with the 

surface. Before the recorded images are assembled into a 
cornposi t e image f redundant inf ormat ion i s removed from 
MP the images. The composite image can be analyzed, in a 

computer for identif ication of charactex - s . 
15 A drawback with the handheld scanners described 

above la LheiL thei.c. field of view ib xelatively small . In 
order to record a large amount of information, such as 
passages consisting of several lines f a user must 
therefore move the scanner back and forth across the 
20 surface repeatedly . Moreover, the movement has to follow 
a predetermined path, such as along the lines of text. 

Publication WO 99/57678 discloses a device fox- 
recording information from a substrate. The device may 
operate in two modes , one suaiiner mode/ in which lines ol 
25 text is scanned, and a photograph mode, in which separate 
pictures are taken of a document or an object. 

Publication WO 9S/2044£ discloses a sca.nnex pen, 
which is adapted to be moved over a line of text for 
scanning the text. As the pen moves over the text image, 
30 several pictures are taken of the text* The pictures are 
processed by a computer and assembled or stitched 
together for forming a composite image of the entire line 
of text, which cannot be read by a single picture . The 
HCcuiuer pen can only scan a single line of tex.t at each 
35 time. 

Thus, there is a need for a handheld scanner pen of 
the above-mentioned type which is adapted to scan several 
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linRfi of timet simnl tariP.oi.isly as well as smaller pictures. 

SUMMARY OP THE INVENTION 

An object of the present invention is to provide 
* a finannRr ppn whinh finable fast recording of text in real 
time. 

Another obj^nt nf the invention is to provide a 
scanner pen which may be used at cl diyLcuiue £xom Lhe Le*L 
and may scan several lines of text in a single stroke as 
10 well as discrete pictures. 

These objects are obtained by a method and a device 
for xeuuxdiny iii£uxmdLiuii by iutayiny on a liyhL-seusi tive 
sensor for obtaining at least two images o£ the 
infuxmaLiuii haviny paxLially uverlappiny cjoxiLeiiLy . The 
15 mpthod comprises converting the information in each o£ 

the iuiaye* lu codud xepx^ttixL<±Licjij., uutctpariiiy Ll«* coded 
representation of said images for determining an overlap 
posiLiun, and dbseuiblyixm the imaM^s Lo form a uutupoyile 
image. The coded representation may be a character code, 
20 yuch as ASCII. AlLex\uaLively , the coded representation 
may comprise a division o£ the information inside 
buaxdery, audi aa recLaiiylab, each comprising portions of 
the information, such as words included in said 
information. Thereafter, the composite imaqe may be 
25 converted to a character code format, such as ASCII-code. 
Alternatively , each iiuaye may be separetly converted into 
character code format, such as ASCII, before aesemblying. 

The method may further comprise determining 
structures in each of said images, such as direction of 
30 lines or text line directions in each imaqe. This may be 
accomplished by means of a Hough transformation of each 
imaqe. This information may be used £or adjusting the 
rotational position and/or perspective of each image in 
dependence of the direction of lines. The information may 
35 also be used for the division of the image in 
reactcuiyleti . 

A concept of the present invention is to record a 
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plu-ral it.y of im^g^R of r text r said images overlapping 
each other, each image comprising several lines of text, 
Subsequently, OCR (Optical Character Recognition) is 
carried out as regards the recorded images, resulting in 
3 fifths of nh?3T^nt.Rrf=! . Thp. RRtR of nharacters may contain a 
number of ohaiaolers ixxdicaLixxy "end u£ lixxe" if the LexL 
comprises several lines. Then the sets of characters are 
assembled utiiixy Llits ulxaxacLera in Llxe bely of cjlioLt"«*uLers . 
An advantage of this mode of operation is that relatively 
10 large images can be recorded at a time, without the 
assembling of the images being cumbersome since the 
images are converted into character uudeb befuxe beixxy 
vjp assembled. The effective resolution is small in a set of 

(if elxaxacterb compared Lu a bit -map imaye, thus saviuy 

15 computing power. The effective resolution in the set of 

didrciuLfcJX-y iy a yixxyle character. Thus, assembling in two 

s 

M: dimensions may be possible with the present invention in 

H J a handheld device. 

J71 The lines of text in two subsequent images do not 

fi 20 xxeceaaarily cuixxcide ixx Llie vexLiual direcLiuxx. The first 

M 1 line in a first recorded image may correspond to the 

second line in a second recorded image. However, the 
assembling will adjust the vertical position so that 
coxxect assembling is obtained. 
25 By a set of characters is meant a plurality of 

characters, from which the relative positions of the 
characters can be determined. The set of characters may 
be a strinq of characters comprising characters for blank 
and end of line. 
30 A device for recordinq a next imaqes the text on a 

light-sensitive sensor with a two-dimensional sensor 
surface, which sensor is intended for diqital recordinq 
of images of the text, said images having partly 
overlapping contents. The device is characterized in that 
35 it is adapted to convert at least two of the images 

xecuxded by Llxe sexxsux ixxLu a t»eL u£ didXduLttxa each, 
comprising a plurality of characters, by means of 
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character recogni ti on . The devi ce i & adapted to 
subsequently assemble the sets of characters with the aid 
of the characters in the sets of characters . 

By carrying out character recognition before 
S flRRPimhl ing t.h& iinflgRR, t.hfi opRrstion of ^RRprnbl i ng large 
images aL pi^el level can be omitted. Moreover/ there is 
less risk that a character would not be recogrH 5:ed owing 
Lo pour absemblixxy, which may be the case when diyiLal 
images are assembled in pixel level and then character 
^ 10 recognition is carried out in an area that is overlapped 

by both images that are assembled. Then, the character 
P may be distorted if the assembling is not carried out 

•J^j properly, which rnay results in that the character is not 

pi xeuoyxiized ixi Lhe uhdiduLei xecoyxxiLiuxx yiuudbti . 

frj 15 According to this invention, the character recognition 

process take advantage, of the oriqinal crualiLy o£ the 
! :: image for character recognition. By first converting the 

y : iuiaqea ixxLu aeLa uf chai.au Lex a, Lhe ddudl ctssemblinq uE 

H* the sets of characters may be quick since the number of 

!j: 20 charauLext* ib considerably smaller than the number of 

pixels in the recorded images. 

Moreover, by OCR interpretation of each imaqe before 
the assembly thereof, a plurality of OCR interpretations 
_ of che same character will be obtained, one for each 

25 image where the character is included, and the 
IxiLexpreLaLiuii which qivea bin lie a L xecuqxiiLiuxi 
probability can be selected. 

Alternatively, it is possible Lo assemble Lhe sets 
of characters using words in the set of characters* Thus, 
30 entire words in one of tiie sets of characters that are to 
be assembled are compared with words in the other of the 
sets of characters that are to be assembled, when 

assembling words, it may be required to compare each 
individual character. 
35 By word is meant a plurality of characters which 

also includes special characters. The special characters 
are, for example, blank, full stop, comma or end* of line. 
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Th« method may nnmpri rp. finding t.h« 1in«R of f.iwt in 
the recorded images, to find start and end of words along 
the Lines of text and to select which of the recorded 
images are to be converted into sets of characters with 
S fchfi aid of the* i rifinf.i f i fid afcarf. and end of the words? in 
the recorded images, so that only images with the 
■n RnRR 9ary information sr« nonvp.rte.d into set* of 
characL«xa ox LhaL images wiLh dupliuaLe information may 
be discarded. By identifying start and end of the words 
w 10 along the lines of text, it will be possible to make a 

K rough assembling of the images without first making 

2 optical character recognition. When the start and end of 

the words have been identified, the recorded images are 
corrected as regards rotation and perspective. By making 
15 a rough assembling of the images, it will be possible to 
find out how the images overlap each other. The imayes 
which contain only information that is available 
completely in other images then need not be converted 
. into sets of characters. 

j7| 20 SLarL and end of words along the lines of text may 

5 be identified by searching, in each pixel along a line 

M : Lhrouyh the lines ol LexL , fur Lhe number of daxk pixels 

a predetermined number of pixels up and down from the 
line of text. An end o£ a word being defined ay i£ there 
25 are no dark pixels in a predetermined number of pixels 

above and btiluw Lhe line of Le-*L, i.e. Lhere are blank*. 
To manage italics one may alternatively search along an 
oblique line. IL is, of course, possible Lu search lor 
white pixels, instead of dark pixels, if the text should 
30 be bxiyhter than the backyround, i.e. inverted. 

The images may be converted into binary images, i.e. 
imaqes containinq merely black and white, since this 
facilitates the continued image processing. 

Moreover, the method may comprise finding the lines 
35 of text in the recorded images using the Hough 

transformation of the recorded images. The Hough 
transformation can briefly be described as follows. There 
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is an infinite number of straight lines extending through 
a point in an XY plane. The equation of the straight line 
for each of these lines can be expressed with two 
parameters. If the parameters of the individual lixxey ax'e 
5 plotted i n a diagram, a curve i .8 obtained which 

corresponds to the Hough transform of the puixit . In thiy 
way, it is possible to plot curves for any of the points 
ixx the XY plaxie. The Houyh transform of two different 
points will intersect In a point, which corresponds to 
^ 10 the eyu<sitiuxx u£ the aLidiylit line extexxdixiy throuyh the 

P« two points. If the Hough transforms for all the dark 

Q pijiela ixx <a xeuoxded ittu^ye «are plutted, a larye number of 

''*Jjj intersections between the different Hough transforms will 

be obtained* However, there is a maximum number of 
IS intersections for lines following the lines of text. 

The device may comprise a memory adapted to store 
the recorded images, which are to be converted into sets 
y : of ulidxciotera, ixx the memory , axxd to convert the stored 

Mi images into sets of characters after completion of the 

W 20 recording of the images. By only storing the recorded 

r? images which are to be converted into sets of characters, 

the memory space which is required for storing recorded 
images is minimized- By converting the stored images into 
^ sets of characters after the recording of imaqes has been 

25 completed, it is not necessary to place high demands on 
the speed o£ the optical character recognition, which 
would be the case if it were to be carried out while 
images are being x*ecox*ded. 

The device is advantageously designed in such a man- 
30 aer that a user can hold it by hand and at a distance 
from a substrate to record text on the substrate; 

The device may be adapted to correct the images for 
rotation before they are converted into sets of 
characters- In the case where the lines of text in the 
35 recorded images have been identified, the correction for 
rotation can be carried out in a relatively simple way. 
However, some optical character recognition programs can 
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process also rotated images, in which case the rotation 
is not required. 

Correspondingly, the. device may correct thp. images 
for perspective before they are converted into s«ts of 
5 characters since the optical character recognition may bp. 
facilitated if all letters have the same size in the 
images . 

The device may be designed as a reading head which 
is connected to a calculating unit in which the recorded 
10 images are processed. 

The device may b«= adapted to aRsemhle the sets of 
characters by comparing the sets o£ characters in pairs t 
'\.U1 the sets ot characters being compared in a number of 

If: relative positions displaced relative to each other, 

m \$ Thus, a first character in a first set of characters is 

42 compared with characters in the second set of characters 

; until correspondence is achieved or until the first 

y. character in the first set of characters has been 

compared with all characters in the second set of char- 
sfel 20 acters. Subsequently, the correspondence of the second 

Characters in the first set of characters is compared 
with the characters in the second set of characters . By 
making the comparison tor a large nnmher of different 
relative positions, a plurality of total numbers of 
points can bp obtained, the total number of points 
reflecting the correspondence between the two sets of 
characters for the specific position. In this way, an 
optimum relative position can be obtained > 

The device may be adapted to store the recorded 
30 images that are to be converted into sets of characters 
along with a ^Rrial number indicating in whinh order the 
images have been recorded, and to assemble the sets of 
characters with the aid of the serial nnmher for the 
images corresponding to the sets of characters. 
'A!) Especially in the oasp. where first all images are record- 
ed and not until then the character recognition and the 
assembling are. begun, the serial numbers may he used for 
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the recorded images since then a large number ot images 
are to be assembled. 

According to a second aspect ot the present inven- 
tion, a method is provided for recording of text on a 
5 substrate, comprising the step of imaging and digitally 
recording images of the text, the images having partly 
overlapping contents. The method is characterized in that 
it comprises the steps of converting at least two of the 
recorded images? into a set ot characters, each with a 
^ 10 plurality of characters, by means of optical character 

recognition, and putting together the sets ot characters 
p with the aid of the characters in the sets of characters . 

v3jn The area renorrted hy thp RfiDRnr may hp. arranged Fin 

that a plurality o£ lines of fcezct are imaged in a 
recorded image. 

According to a third aspect of the present inven- 
I , tion, a cornpisfcer-readabJ e storage medium, in which a 

M* computer program is stored which is adapted to be used 

for r.onvfirsi on of digital images, which are recorded hy 
20 an image sensor/ into text. The storage medium is char- 
ts acterized in that the computer program comprises instruc- 
tions for making the computer receive digital images as 
input pngnalft, convert the digital images into flftf.PS of 
V^/ characters, with a plurality of characters, by means of 

2S nh?37V3nt:Rr -rp.nngni ti on , and put together the sets .of char- 
acters with the aid of the characters in the sets of 
nhsranter.g. 

Further objects, features and advantages of the 
invention will appear from the following detailed 
30 description of embodiments of the invention with 
rflfPTPnce to f.hft drawings. 

ttttTFn? DF.RH'RTPTTO'M OP TTTC DRAWINGS 

Fig. 1 is a schematic view of a device according to 
35 a tirsst embodiment o± the present invention. 

Fig. 2 is a schematic block diagram of parts of the 
fimhoriiment of Fig 1 . 
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Pig. 3 is a schematic view of images of text on a 
sheet of paper, which are recorded according to the 
invention. 

Figs, 4a and 4b a ore diagrams, illustrating the 
h prinniplR of the Hough transformation. 

Fig. 5 is a diagram, which shows maximum points for 
the Hough transform nf two diffpvrftnt imagRfi. 

Fig. 6 is a diagram and a histogram for illustrating 
the detection of start and end points of words. 
10 Figs. 7a. and 7b are diagrams for illustrating the 

H 1 division of the text images into words . 

S Fig. 8 is a diagram of two images converted to 

characters for assembling. 
*P Fig. 3 is a diagram for illustrating lines of text 

15 in a recorded image. 

Fig. 10 is a £low chaxt u£ the operation u£ a com- 
puter program according to the invention. 

Fiq, 11a, lib and 11c are shematic representations 
of a text, division thereof in rectangles and display on 
20 a small display. 

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION 

Fig. 1 discloses a scanner pen, comprises a casing 1 
havinq approximately the same shape as a conventional 
25 highlighter. In one short side of the casing there is an 
opening 2, which Its intended to be directed at an area uii 
a substrate which a user wants to image. The information 
Udrrier can be a sheet of papez . 

The casing 1 essentially contains an optics part 3, 
30 an electronic circuitry part 4 and a puwex* supply part 5- 
The optics part 3 comprises a lens system 7 , light - 
emitting diodes 6, and an optical sensor 8 constituting 
an interface with the electronic circuitry part 4 . The 
light -emitting diodes 6 may be used to increase the 
35 illumination. 

The light-sensitive optical sensor 8 may be a two- 
dimensional CMOS unit or CCD unit (CCH = rhargR flmipl rc) 
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Device) with a built-in AD converter. Such sensors are 
commercially available . The sensor 8 may be mounted on a 
printed circuit hoard 11. 

The power supply to the device is obtained from a 
5 lnat.1-.fvry 12 which ia mount ad In a separate compartment 13 
in the casing. 

Fig. 2 is a block RohemR of the electronic circuitry 
part 4, which comprises a processor 20 which, via a bua 
21 , is connected to a read-only memory ROM 22 , in which 
10 the program of the processor is stored, to a write/read 
M: memory RAM 23, which is the work memory of the processor 

^ and in which the linages fx cm Lhe senaui eis well as 

characters that are interpreted from the recorded images 
are stuxed, Lu a contxol logic unit 24 eiud to Lhe awuaur 

The control luyic unit 24 is connected to a number 
of peripherals, such as a display 25 mounted in the 
casing, an IR transceiver or short-ranqe radio link 26 
for transferring information to/from an external . 
20 coiuputex', but Lena 27 by means o£ which Lhe user can 

control the device, and an operation indicating device 28 
conaiutiuy ci a second set of 1 iyht- etui t Ling diodes which 
may indicate whether recording occurs or not and other 
opexatiuy conditions* The control lou.ic unit 24 yexxerates 
25 control signals to the memories, the sensor and the 
periphexals. The control loqic unit disc luanaqes the 
generation and the prioritization of interrupts to the 
processor. The buttoxis 27, the txcinsceiver 26, the 
display 25 and the light -emit ting diodes £ are controlled 
30 by the processor by wx'itiny ciud reading in the records of 
the control logic unit. The buttons 27 generate 
iutexxupts to the processor 20 when activated. 

The function of the device will now be described, A 
sheet of paper 9 is provided with a plurality of .lines of. 
35 printed text 10 as shown in Pig, 3. When a user activates 
the scanner pen by means of the buttons 27 and passes it 
across the sheet of paper with the opening 2 directed 
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towards the sheet of paper, three images 14, lb, 16 are 
recorded. Each of the images 14, 15, 16 io cxpoocd to OCR 
processing and the text ot the images is converted into 
sets of characters as illustrated in Fig- 0- Subsequently 
5 the sets ot characters are assembled or stitched so as to 
form a complete text. As shown in Fig. 3, the images 14, 
15, 16 may be rotated in relation to each other. Thus, 
a first image 14 is turned or rotated in relation to a 
second image 15, whir.h in turn is rotated in relation to 

^\ 10 a third image 1G. 

Tn o-rd^r to optimise* the optical character recogni- 
tion in the recorded images 14, 15, 10 it is advantageous 
to know the ortentatinn ot thR linRS of tRxt in thR 
image. Therefore, the orientation of the lines of text is 
IS rip.tftntfid hftforft the character recognition is carried out. 
The detection of the orientation of the lines of 
f.Rvt may hR carried out using thR Hough transformation. 

Referring to Figs 4 and 5, the Hough transformation 
will now bR gRwally dRsnrihRd. Fig. 4a fihnwfi fivR 
20 points in a plane with the coordinate axes X and Y. Pig 
4h shows the Hough transform o± the five points in Pig. 
4a. A first point 18a has a first Hough transform curve 
19s which dRRnrihRR all thR straight "Mtirs Rxt«nding 
through the first point 18a in Fig, 4a as a function of 
'1% thR psramRtRrs H and p whRrt* 9 is thR angl « of the 

straight lines through the point and p is the distance of 
the straight lines from origin. The Hough transform 
curves have a sinusoidal shape. Correspondingly, the 
second 18b, third 18c, fourth ISd and fifth 16e points 
30 have <x second 19b, third 13c, fourth 19d and fifth I9e 
Hough transform cutvr . ThR sRnnnd 1 9h, thi-rd 1 9r. and 
fourLh 19d Hough transform curves in Fig. 4b intersect in 
a point. This point in Fig. 4b norrRspnnds tn a straight 
line in Fig. 4a extending through both the second point 
■*S IRh and thR third point 1 fir- and the fourth point 18cL 
Fig, 9 shows the second recorded image 15 from 
Pig. 3, which consists of a plurality of pixels in whir.h 
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lines of text are to be found. For each black pixel in 
the second recorded image 15, a Hough, transform curve is 
calnnlated. Thn Hough transforms ar^ inserted in one and 
the s ame di agr am . 
.*> Fig. 9 shows a first line 32 and a second line 33. 

Lines having approximately the same direction as the 
first, linp 32 wi I I intersect a larger number of points 
than lines having approximately the same inclination as 
the second line 33 >sinne the darfc pixels in the recorded 
10 image 13 are positioned along lines of text having 

approximately the same dirp.nhion as th<=» first, line 32. 

Pig. 5 illustrates that the maximum number of 
intersections can be used to determine the orientation 
of the lines of text. The circles 73 correspond to points 
15 in the Hough trariflfnrm diagram whevre .q^VRral tfough 

Lid.nfc$£oxiu uuives of points in Fig, 9 intersect, i.e* 
maximum of intersect ions . The nirr.l 73 corresponds to 
the image 15 and are positioned along a straight line 34, 
Prom the distance between the circles 73, the distance 
jy 20 between the lines of text can be determined. The position 

13 of the intersecting line 34 along the 8 axis indicates 

the luLdLiuii of the recorded image. The second line 35 in 
Fig. 5 corresponds to the image 14 and pretends through a 
pluxctliLy u£ tiieu4j.tttum indicated by crosses 3 6 in the 
25 diagram. The inclination of the second line 3 5 indi nates 
that the image has a perspective , i.e. the lines of text 
N have different rotations. Also the sligth different 

distances between the crosses 36 indicate that the image 
has a perspective with larger distances between the lines 
30 clL Lhe luw&x. puxLion of the line 36, The displacement of 
the second intersecting line 35 in relation to the first 
line aloncj the 0 a.xis ixidiudLea LhaL Llie lines of text 
are rotated in the recorded image. By means of this 
informed Lion, Llie iuiaye may be adjusted for perspective 
IS and rotated, for ftxsmpl p. to th« horizontal direction 
which is the same direction as image 15. As appears, 
image 15 corresponding to Tin* 34 need no adjustment. 
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while image. 14 corresponding to line 35 needs adjustment 
of the perspective to make line 3 6 vertical and with 
approximately equidistant crosses 36 and adjustment as to 
the rotational posit ion , to move line 35 to the same 
5 angle 0 as image lb, which may correspond to zero angle. 

After identification of the lines of text, an iden- 
tification of the start and end parts of the words in the 
recorded image is carried out- Pig- 6 indicates how the 
iRt-.tR-r "e" ^7 is detected. As shown .in Pig. 6, the number 
10 of dark pixels 7 are counted in the vertical direction 
perpendi mil ar to the line 38, which may he calculated as 
described above. The number of dark pixels is zero up to 
\Jfl the start 39 ot thR letter "e" and will again he zero at 

J; the end 4 0 of the letter "e" . When the number of dark 

£Q 15 pixels has been *ero for a predetermined period, this is 

*P detected as the end of a word* 

:.. With rftfp.rp.nnp. to Fig. 7, the words are indicated as 

ju- rectangles, the start 41 of the rectangles indicating the 

M start of a word an<i the p.nri 42 of the rectangles 

{A*. 20 indicating the end of a word. Fig. 7a corresponds to a 

2 tirst recorded image 14 and Fig. 7h norrp.Rpnnds to a 

second X'ecox'ded image 15 . A length of a first word 43 
in Fig. 7a has correspondence in a length of a second 
word 44 in Fig. 7b- Correspondingly, a length of a third 
7.5 word 45 in Fig. 7a has correspondence in a length of" a 

fourth word 46 in Fig, 7b, By matching the two images, it 
is possible to find out how the recorded images overlap 
each other, by only* using the graphical information of 
the length of each word. 
30 Thus, by using the length of the words for each 

line, it is possible to carry out. a rough puthing- 
LuyeLhex* ui tsLitching or assembling of the two images* 
The images are roughly assembled so that a sequence of 
word lengths in the first recorded image corresponds to a 
sequence n f word lengths in the second recorded image. 
The word lengths along different lines in the first 
recorded image should thus correspond to word lengths 
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along corresponding lines in the second recorded image* 
In this way, it is possible to determine how the imagca 
arp. diRplsnpd relative to each other. 

Pig, 3 shows how this may be used to sort out 
5 unnecessary images so Chat optical character recognition 
need not be carried out as regards all imaged that are 
recorded. A first image 47 and a second image 49 com- 
pletely overlap the area which is covered by a third 
rftcorriftri image 48 which is indicated by dashed lines. By 
^ 10 using the method as described in connection with Fig. 7, 

y f . the third rp.nordp.d image 4fi can he completely omitted, 

C| without optical character recognition being carried out. 

-W Of course, Fig. ^ is only schematic, since r.he images 

III 

*jp often overlap each other to a much larger extent. Indeed, 

fy 15 it the scanner pen is held approximately still, all 

images overlap more or less, and a substantial saving of 
nnmpnting pnwfir may he ohtaineri by omitting images with 
duplicate information. On the other hand, during normal 
scanning operation, b partial ovrt! ap may hp. used f or 
pt 20 sorting out errors in the optical character recognition 

P*j prongs or thp assPTrihling pronefis, in which case 

ft! overlapping images are not omitted or discarded. 

Another method ot assembling the images is disclosed 
in Fig. 8/ which shows a first set of characters 50 with 
75 a plurality of chsrantRrs fin corresponding to the first, 
recorded image 47 in Pig. 3 and a second set of charac- 
^ ters 51 with a plurality of characters 61 corresponding 

to the second recorded image 49 in Pig. 3. The text in 
the first set of characters and the second set of char- 
30 acters follows the direction of the lines of text C2, 
which may have* heen obtained by the Hough transform 
process described above. The first set of characters GO 
and the second set of characters 51 are put together or 
assembled by comparing characters in the two sets of 
characters. Firstly, the tirst character 52 in the tirst 
set of characters is compared with each of the characters 
in the second set ot characters 51. The operation 



pi 

(§5* 
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proceeds correspondingly for the second character 63 and 
the third character 64 in the first set of characters SO. 
Good correspondence is obtained when the characters in 
the word "pkillGd" 53 in the second line of the first set 

5 of characters is compared with the word "skilled" 54 in 
the first line in the second set of charactcra 51. Since 
a word can appear in many positions in a text, one starts 
from the first correspondence found and then compares the 
rest of the text for this position, a total number of 

10 pointc being obtained which indicates how well the two 
sets of characters correspond for this position. 
Subsequently this step is repeated for the next position 
where correspondence is obtained. Finally, the position 
is selected in which the total number of points indicates 

15 the best correspondence- in Fig. 8, the text is in 

English but a person skilled in the art undcrotando that 
the text could just as well be in any other language and 
that the text in Pig- 8 ic only uced to illustrate the 
function of the device. The image may also comprise 

20 symbols, such as mathematical expressions, or even line 
art, to a limited extent. 

The two assembling methods may be combined , so that 
the images are first compared in a rough manner and then 
on character level. In this way, it will be possible to 

25 carry out assmebling in two dimensions with limited 

computing power. If the reading device ic firot moved to 
the right like in Fig. 3 and then down and to the left, a 
larger surface is obtained which is to be assembled in 
two dimensions. Ry determining the mutual relationship ot 

30 the partial images by rough assembling and then on 

character level, it is relatively easy to obtain both 
horizontal and vertical assembling. The characteristic 
that the lines determine the vertical position with great 
accuracy results in the possibility of alignment. 

35 Each assembling method may alternatively be used 

separately. If the method using length of words is used 
separately, the images may be assembled into a composite 
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image before final conversion of the images zo characters 
by optical character recognition, This method may be uccd 
when the images comprises other matter than text, such as 
small pictures or symbols. In this case, the pymbolc or 

5 small pictures will be handled as if they were words and 
be included in a rectangle. The assembling process will 
work, provider* that the pictures or symbols are 
sufficiently small. However, the OCR process will fail to 
rfinngni7.fi the rer.tang 1 ft as characters, and then the pixel 

10 representation may be used instead. Other types of 

hoarders may bfi used than rectangles, such as an area 
delimited by more than four straight lines or even curved 
lines . 

If the assembling is carried out after the 
15 conversion nf eanh image to character code such as ASCII, 
small picLures and symbols may be handled separately, at 
least if they are surrounded by text matter in each 



N : image , 



As appears from Fig. 7 r the imagp.fi may be processes 
20 by division of the image in boarders, such as rectangles. 
Each rectangle comprises a complete word or a Ryrnhol or a 
picture surrounded by white areas. Several pictures are 
compared for finding a snnneRRi on nf r^ntsng] ftfi which 
cuzieapuudtf Lo Lhe wanted text. When overlapping 
25 positions have been determined, fisnh rp.ntang'1 e is given a 
succession number, comprising the line number and the 
word number on that line. Thne, in Fig*- 7a and 7ft, the 
word 41 obtains designation 1:1, the next word 1:2, and 
the following word 1:3. On the second line, the first 
30 work obLaixia Lht* designation 2:1, the second word 2;2, 
the third word 2:3 and the fourth word 2?*- 
Correspondingly , on the Lhird line, Lhe first word 
obtains designation the second word t>z?., the third 

word 3:3 and the fourth word 3:4- It is now recognized 
that the. fourth word on the second row, 2:4 corresponds 
to the second word on line 2 of Pig. 7b , which means that 
the first word at the second line of Fig. 7b obtains 
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designation 2:3, the second, word 2:4 (corresponding to 
the fourth word of line 2 of Pig. 7a)/ the third word 2:5 
and the fourth word 2:6. The same goes for the third line 
of Pig- 7b, in which the firot word obtainc designation 
5 3:3, the second word 3:4, the third word 3:5 and the 

fourth word 3=6- The words arc then arranged in a row of 
words forming a complete line. 

It can now be seen that several words arc duplicated 
in the two pictures, namely word 2:4 and word 3:4. There 
V 10 arc further words which arc duplicated on the other 

M: lines. These duplications may be omitted and replaced by 

2 a single word. Else the duplication may bo kept and uocd 

\JK for increasing the OCR interpretation accuracy. 

4S Finally, the words arc OCR processed in the right 

5^ 15 order to obtain the desired text. 

% If any rectangle ia larger in the vertical direction 

/ than a single line, it may obtain designation like 2,3:6, 

if it occupies lines two and three- In this way, larger 
objects? such as pictures or symbols may be handled. 
20 There are a number of cases in which partial words 

are included in Fig. va but included in full in Fig. 7b, 
such as word 1:3. In this case, the longest version of 
the word is used for interpretation, if there is a doubt 
if this is correct, all fragments may be used to recreate 
25 the complete word. 

in this way, the images arc assembled on a word 
basis starting from the pixel representation and dividing 
the image inside boarders, such as rectangles which arc 
compared for the best overlapping position. Then, 
30 duplicate information is omitted or used for further 
accuracy and then the words are arranged in the right 
order and finally converted to ASCII -code. 

Pictures or symbols which may not be recognised by 
the OCR program, may be maintained in pixel format and 
35 displayed as such in the final picture. For increased 
oafcty, aloo at leant one version of the words may be 
kept in pixel format in parallel with the OCR-version, 



left 
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Bfipwr.ially if tih* OCT* program inriioat.Rs a poor qiia'M ty of 

processing, 

Every new image is compared with previous 
information in order to find out its orientation therein. 
5 Thus, ear.h image is processed both in the vertical 

direction as well as in the horizontal direction, both 
forwards and backwards. Thus, it is possible to scan in 
two dimensions by the present invention. This is possible 
because of the division of the image in a coded 
^ 10 representation which is less time consuming to process, 

H' either in the form of rectangles or similar or in the 

J;!! form of ASCII code. 

It may be o± interest to have some kind ot teed -back 
that the desired information is gathered ♦ This may be 
15 annrnnplighftrt by displaying the* asRP.mblftd information at 
the scieeu. However, since the display at a handheld 
device is rather small, another method would be to 
display the lines as a succession of pixels at the 
J display, in which one pixel approximately corresponds to 

20 a single character. Then, the characters will form words 
and the layout, of words would give a visual indication of 
the scanned surface. If the processing comprises division 
into rectangle*, thftRR rpr.tangl rr may bR indicated a* 
Lhey axe assembled. 
75 Figs 11a, lib and 11c show how this may be 

accomplished. Fig. 11a is a text that is to be scanned* 
Pig. lib is the division of this text in rectangles. 
Finally, Fig. 11c is the representation of the rectangles 
at a small display, in which rectangles are indicated in 
30 black pixels and spaces between the rectangles as gray 
pivels. From Fig. 11 n it nan b^ rp.rti that rohir 
infoxuiaLiuLL is missing as indicated by white pixels 29. 
The user then directs his scanner pen towards the area 
missing until the display indicates that all areas are 
H5 fully novprftfi by at Ipsast nnp. images. Finally, the. noml Rtps 
image is converted to ASCII, if that has not been done 
ftarlif^r in th« pronpsft. 
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if the assembling is done by using the coded 
representation in the nature of ASCII code, each decoded 
character is displayed as a black dot an the display 
screen, while spaces are displayed as grey dots > Any 
5 white doc will indicate that information is missing, as 

described above - 

According to the present invention, it is required 

that the information at least partially is positioned 
along identifiable lines, if the device is passed across 
Ww ' 10 a photograph or some other surface which is not divided 

y, into lines, this can easily be recorded by the processor 

O in the device, and this part of the image may be 

vjg! di scarrfc&d ot stored separately as an picture or 

y i 

jr; photograph. If said surface is completely or at least 

15 partially surrounded by lines, it would be possible to 
handle the situation via the invention, as soon as at 
least a portion of a line is included in every image, 
jsfc Pig- 10 is a flow chart of the operation of a com 

H : puter program according to the invention. The computer 

20 program is adapted to be executed in the electronic cir- 
p mi i try part 4 of the device, in a first step bb, digital 

S 5 * images are received from the sensor 0 * In a second step 

56, the digital images are converted into strings of 
■<w characters using character recognition . In a third step 

25 57, the strings of characters are assembled or put 
together . 

Tt is nnt necessary to carry out the rough putting- 
together of the recorded images as described in 
nnnnAntion with Pig. 7, and optical character recognition 

30 can be carried out directly as regards the recorded 

images- Moreover, the orientation ot the lines of text 
need not be identified in the recorded images if an 
optical character recognition algorithm is used which is 
able to identify characters also when the lines of text 

35 are rotated. 

A person skilled in the art rcalizca that the inven- 
tion is not limited to the embodiments shown and that 
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many modi finafci oris are. fRasihlR within th*=> <=inopf=> of the 
invention* The invention is only limited by the appended 
patent nl^imtf. 
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